Automated Mapping of the MapReduce Pattern onto Parallel Computing Platforms

The MapReduce pattern can be found in many important applications, and can be exploited to significantly improve system parallelism. Unlike previous work, in which designers explicitly specify how to exploit the pattern, we develop a compilation approach for mapping applications with the MapReduce pattern automatically onto Field-Programmable Gate Array (FPGA) based parallel computing platforms. We formulate the problem of mapping the MapReduce pattern to hardware as a geometric programming model; this model exploits loop-level parallelism and pipelining to give an optimal implementation on given hardware resources. The approach is capable of handling single and multiple nested MapReduce patterns. Furthermore, we explore important variations of MapReduce, such as using a linear structure rather than a tree structure for merging intermediate results generated in parallel. Results for six benchmarks show that our approach can find performance-optimal designs in the design space, improving system performance by up to 170 times compared to the initial designs on the target platform. Q. Liu · T. Todman (B) · W. Luk Department of Computing, Imperial College London, London SW7 2AZ, UK e-mail: [email protected] Q. Liu e-mail: [email protected] W. Luk e-mail: [email protected] G. A. Constantinides Department of Electrical Engineering, Imperial College London, London SW7 2AZ, UK e-mail: [email protected]